Introducing an online training course in Campbell systematic review methods

In April 2020, members of the Campbell Collaboration Methods Group and Campbell leadership met to discuss options for creating flexible training opportunities for Campbell reviewers. It was not a coincidence that this meeting occurred at the beginning of the Covid‐19 pandemic. But in truth, conversations about how Campbell might increase the effectiveness and reach of Campbell training started at least a decade earlier. Training in systematic review methods has always been important to Campbell—we have a Methods sub‐group that is focused on training, and training opportunities have been part of every Campbell annual meeting. In addition, Campbell typically offers one or two standalone workshops sessions a year to outside groups seeking training in systematic review methods. We have never had a good way of evaluating the effectiveness of these one‐off training experiences, and in addition, we worried about the cost and access issues associated with in‐person training. Further, we could not help but notice that when Campbell training sessions are accessible to a broad audience, they tend to be very popular. As an example of this latter point, David Wilson's presentation on effect sizes and basic issues in meta‐analysis, which was part of a training workshop at the Campbell Colloquium in 2011, has been viewed over 49,000 times (Wilson, 2011) as of December 2022. As we investigated options for addressing questions regarding training effectiveness, resource efficiency, and access equity, we searched for platforms that would allow us to host training materials in an online environment, that could be accessed at little or no cost to users, and that have tools for assessing learning. Ultimately, we chose to work with the Open Learning Initiative (OLI) at Carnegie Mellon University (https://oli.cmu.edu/). Over the course of the next 30 months, a team of seven individuals with expertise in systematic reviews and meta‐analysis, led by Jeff Valentine, Julia Littell, and Sarah Young, plus Greg Bunyea, a learning engineer from OLI, devoted thousands of hours to create a course titled: Systematic reviews and meta‐analysis: A Campbell Collaboration online course (Valentine et al., 2022). We were ably assisted in this work by Mark Englebert, Jennifer Hanratty, Terri Pigott, and Zahra Premji. The remainder of this essay is devoted to describing the scope of the course, its primary audience, its organization, and the principles we adopted during development.


Introducing an online training course in Campbell systematic review methods
In April 2020, members of the Campbell Collaboration Methods Group and Campbell leadership met to discuss options for creating flexible training opportunities for Campbell reviewers.
It was not a coincidence that this meeting occurred at the beginning of the Covid-19 pandemic. But in truth, conversations about how Campbell might increase the effectiveness and reach of Campbell training started at least a decade earlier. Training in systematic review methods has always been important to Campbell-we have a Methods sub-group that is focused on training, and training opportunities have been part of every Campbell annual meeting. In addition, Campbell typically offers one or two standalone workshops sessions a year to outside groups seeking training in systematic review methods. We have never had a good way of evaluating the effectiveness of these one-off training experiences, and in addition, we worried about the cost and access issues associated with in-person training.
Further, we could not help but notice that when Campbell training sessions are accessible to a broad audience, they tend to be very popular. As an example of this latter point, David Wilson's presentation on effect sizes and basic issues in meta-analysis, which was part of a training workshop at the Campbell Colloquium in 2011, has been viewed over 49,000 times (Wilson, 2011) as of December 2022.
As we investigated options for addressing questions regarding training effectiveness, resource efficiency, and access equity, we searched for platforms that would allow us to host training materials in an online environment, that could be accessed at little or no cost to users, and that have tools for assessing learning. Ultimately, we chose to work with the Open Learning Initiative (OLI) at Carnegie Mellon University (https://oli.cmu.edu/). Over the course of the next 30 months, a team of seven individuals with expertise in systematic reviews and meta-analysis, led by Jeff Valentine, Julia Littell, and Sarah Young, plus Greg Bunyea, a learning engineer from OLI, devoted thousands of hours to create a course titled: Systematic reviews and meta-analysis: A Campbell Collaboration online course (Valentine et al., 2022). We were ably assisted in this work by Mark Englebert, Jennifer Hanratty, Terri Pigott, and Zahra Premji. The remainder of this essay is devoted to describing the scope of the course, its primary audience, its organization, and the principles we adopted during development.

| COURSE PURPOSE AND ORGANIZATION
Systematic reviews and meta-analysis: A Campbell Collaboration online course is aimed at Campbell reviewers and others who want to learn how to find, assess, and synthesize the results of relevant studies to inform policy, practice, and future research or in other words, people who want to learn how to conduct systematic reviews and metaanalysis. We assume that learners will have prior graduate training in research methodology and statistics. We designed the course to be suitable for both classroom and independent learning and view it as the equivalent to a textbook or an introductory, graduate-level course in systematic reviewing. It should also work well as an adjunct to in-person workshop training. The content on systematic review methods is relevant to systematic reviews regardless of the nature of the specific research hypotheses being investigated, but the content addressing synthesis methods is focused on the synthesis of quantitative data (meta-analysis).